Eye tracking head mounted display device

ABSTRACT

This document relates to head mounted display devices. One example can include a housing configured to be positioned relative to a head and eye of a user and a transparent visual assembly positioned by the housing in front of the user&#39;s eye and comprising multiple eye tracking illuminators distributed across the transparent visual assembly and configured to emit non-visible light and multiple eye tracking detectors distributed across the transparent visual assembly and configured to detect the non-visible light reflected back from the eye of the user.

BACKGROUND

Head mounted display devices can enable users to experience immersivevirtual reality scenarios and/or augmented reality scenarios. Suchtechnology may be incorporated into a device in the form of eyeglasses,goggles, a helmet, a visor, or some other type of head-mounted display(HMD) device or eyewear. In order for the HMD device to be comfortablefor any length of time, the head mounted display should be positionedrelatively closely to the user's face (e.g., eyes) and should berelatively light weight. Despite these constraints, the HMD deviceshould be able to perform multiple functionalities, such as imagegeneration, eye tracking, and/or 3D sensing of the environment. Thepresent concepts can address these and/or other issues.

BRIEF DESCRIPTION OF THE DRAWINGS

The Detailed Description is described with reference to the accompanyingfigures. In the figures, the left-most digit(s) of a reference numberidentifies the figure in which the reference number first appears. Theuse of similar reference numbers in different instances in thedescription and the figures may indicate similar or identical items. Insome figures where multiple instances of an element are illustrated, notall instances are designated to avoid clutter on the drawing page.

FIG. 1A illustrates a perspective view of an example HMD device that isconsistent with some implementations of the present concepts.

FIGS. 1B, 2, 3, 4, 5, 6, 7A-7D, and 8-16 illustrate elevational views ofexample HMD devices that are consistent with some implementations of thepresent concepts.

FIG. 17 illustrates example methods or techniques that are consistentwith some implementations of the present concepts.

DETAILED DESCRIPTION Overview

Head-mounted display (HMD) devices can present virtual content to a userin a virtual reality scenario and/or an augmented reality scenario. Aprimary function of the HMD device is to display images at an ‘eye box’for perception by the user. While the display function is a centralfunction of the HMD device, other functions, such as sensing theenvironment via depth sensing (e.g., 3D sensing) and eye tracking tounderstand the user's interaction within the environment can be valuablefunctions that contribute to the overall quality of the user experience.Traditionally, 3D sensing and eye tracking have been accomplished withdedicated components positioned outside of the user's field of view(FoV).

The present concepts can accomplish the eye tracking and/or 3D sensingwithin the FoV of the HMD device. The concepts can include multiple waysthat 3D sensing, eye tracking, and/or image generation can be enhanced,simplified, and/or reduced in cost by employing a distributed anddispersed arrangement of electronic components and/or optical componentson a visual assembly. The electronic components can be small enough thatthey are imperceptible to the user. The visual assembly can betransparent to visible light despite the distributed and dispersedarrangement of electronic components and/or optical components on thevisual assembly. Utilizing multiple electronic components dispersed anddistributed across the FoV can offer several advantages over traditionaldesigns. These and other aspects are discussed below.

Introductory FIGS. 1A and 1B collectively depict an example HMD device100 which can implement the present concepts. HMD device 100 can includea housing 102 that can orient a visual assembly 104 relative to a user106. In some cases, the visual assembly 104 can include an electricallayer 108. In some implementations, the visual assembly 104 can betransparent in that it can allow ambient light 110 to pass through andreach an eye box 112 associated with the user's eye 114. The transparentvisual assembly 104 can also include side-by-side electronic components116 distributed on the electrical layer 108. The term side-by-side isused to indicate that the electronic components are positioned adjacentto one another on the electrical layer 108 either abutting or with gapsin between.

The electronic components 116 can perform various light generation andlight detection functions. For instance, electronic components 116(1)and 116(7) can generate non-visible light (shown as dotted lines), suchas infra-red (IR) light that can be directed toward the eye box 112 togain information about the user's eye 114. Electronic component 116(4)can detect the non-visible light reflected from the user's eye to gaininformation about the user's eye. Electronic component 116(3) cangenerate non-visible light (shown as dashed lines), such as infra-red(IR) light that can be directed toward the environment to gaininformation about the environment. Electronic component 116(6) candetect the non-visible light returned from the environment to gaininformation about the environment, such as by 3D sensing/mapping.Electronic components 116(2) and 116(5) can generate visible light(shown as solid lines) that can be directed toward the eye box 112 tocollectively generate a virtual image. These are just some of the typesof example electronic component types that can occur on the electricallayer 108. Other examples are described below relative to FIG. 2 .

As mentioned above, in some implementations ambient light 110 can passthrough the virtual assembly 104 so that the user can see both theactual physical environment and virtual content (e.g., augmentedreality) generated by a subset of the electronic components 116. Eachtype of electronic component 116 can be distributed and dispersed acrossthe electronic layer (e.g., can have neighbors of different electroniccomponent function). This aspect will be described in greater detailbelow relative to FIGS. 2 and 3 . This configuration can be contrastedwith traditional technologies that employ eye tracking and depth sensingcomponents around a periphery of the HMD device, but not in the device'sFoV.

In some virtual reality scenarios, the visual assembly 104 may not betransparent, but the electrical layer can be transparent. For instance,eye tracking electronic components on the electrical layer would notdegrade visual images from a display positioned away from the user inthe visual assembly relative to the electrical layer 108.

Note also that for ease of illustration and for sake of brevity, FIG. 1Bas well as some of the subsequent FIGURES show only one of the user'seyes and part of the visual assembly 104 in front of the eye. However,the described concepts can be applied to both the left and right eyes bythe HMD device 100.

FIG. 2 shows another example HMD device 100A. (The suffix, such as ‘A’is used relative to HMD device 100A for purposes of distinguishing thisHMD device from HMD device examples above and below. The various HMDdevice examples may have different and/or additional elements and/orsome elements may be different in one implementation compared to otherimplementations.) In this case, the visual assembly 104 can include anoptical layer 202. In this configuration, the optical layer 202 ispositioned away from the eye 114 relative to the electrical layer 108.In other configurations, the optical layer 202 could be positioned onthe opposite side of the electrical layer 108.

The optical layer 202 can include multiple optical components 204 thatcan be positioned side-by-side to one another on the optical layer. Theoptical components 204 can be configured to affect a path of some or allwavelengths of light that encounter an individual optical component. Forinstance, the optical components 204 can be manifest as mirrors and/orlenses. The optical components 204 can work cooperatively with theelectronic components 116 to achieve various functionalities, such aseye tracking, image generation (e.g., RGB display), and/or 3D mapping,among others. Note that the optical components 204 and the electroniccomponents 116 tend to be very small and as such are not drawn to scaleand/or in the numbers that would likely be present on the visualassembly 104, but the illustrated optical components 204 and theelectronic components 116 serve to convey the present concepts.

In this example, electronic component 116(1) and optical component204(1) operate cooperatively to contribute to RGB image generation andthus can be viewed as an RGB display module 206(1). The electroniccomponent 116(1) can entail a red, green, blue (RGB) display (e.g.,pixel cluster), such as a light emitting diode(s) (LED) that isconfigured to emit light in a direction away from the eye 114. In thiscase, the optical component 204(1) can be manifest as a partiallyreflective mirror or a notch filter. A partially reflective mirror canreflect certain wavelengths of light while being transmissive to otherwavelengths of light. Alternatively or additionally, a partiallyreflective mirror can reflect light received at certain angles whilebeing transmissive to other angles. For instance, ambient light 110traveling generally normal to the optical axis may pass through thepartially reflective mirror 502(1). In contrast, the partiallyreflective mirror of optical component 204(1) can reflect the RGB lightfrom the electronic component 116(1) back toward the eye 114. While onlyone RGB or single-color display module is shown, multiple dispersed anddistributed RGB display modules 206 can contribute to the overall imageperceived by the eye 114.

In the illustrated configuration, electronic components 116(3) and116(4) can emit non-visible light for ET purposes. For instance, theelectronic component 116(4) can be an IR LED or array of LEDs. Thisnon-visible light can be emitted in a direction away from the eye andcan be redirected back toward the eye by optical components 204(5) and204(6), respectively that are manifest as partially reflective mirrors(e.g., hot mirrors), for instance. A hot mirror can transmit visiblelight while reflecting non-visible wavelengths, such as IR. Electroniccomponent 116(3) and optical component 204(5) can function as an eyetracking illumination module 208(1) and electronic component 116(4) andoptical component 204(6) can function as eye tracking illuminationmodule 208(2). Note that electronic components 116(4) and 116(5) mayemit the same wavelengths of non-visible light. In other configurations,these electronic components may emit different wavelengths of light fromone another. Potential advantages of this latter configuration aredescribed below relative to FIG. 15 . Electronic component 116(5) canemit non-visible light for 3D mapping purposes and can function as a 3Dmapping or depth map module 212.

Electronic component 116(2) can include a sensor that is sensitive tothe non-visible light. The non-visible light can be emitted by ETillumination modules 208 and reflected back from the user's eye. Thenon-visible light can be received at optical component 204(3), whichredirects the light toward the electronic component 116(2). Thus,electronic component 116(2) and optical component 204(3) can function asan ET camera/sensing/detection module 210(1).

Other electronic components can entail multiple components thatcollectively can both emit non-visible light, such as IR, and sensenon-visible light that is reflected back from objects in theenvironment. For instance, the emitting component can entail an IR LEDor LED array and the detector can entail an IR CMOS sensor, for example.The IR light can be structured light and/or can be sensedstereoscopically (e.g., by multiple detectors) to convey 3D information.These configurations can enable 3D mapping of the environment in frontof the user. In some cases, the electronic component is not paired withan optical component in the optical layer 202 (e.g., does not needfocusing). For instance, the non-visible light can be emitted evenly ina flood pattern that can be effective without redirecting of thenon-visible light that could be provided by an optical component.However, in other implementations, an optical component, such as varioustypes of mirrors and/or lenses, can be employed to affect the lightemitted from the electronic component. In either configuration (e.g.,without or without an optical component) the electronic component can beviewed as contributing to a module configured to achieve afunctionality.

Two of the depth sensing techniques that can be accomplished with thepresent implementations can include time of flight (ToF) techniques andstereo techniques. Time of flight can rely on measuring the time lightneeds to travel from the source (e.g., the IR emitter of electroniccomponent 116(5) to the object and then back to the IR detector/sensor(e.g., camera) of electronic component 116(5). The sensor can measurethe time the light has taken to travel and a value of the distance canbe established. ToF techniques tend to utilize an optical pulse or atrain of pulses. In addition, there is often a desire for the emittedbeam to have a certain profile (this reduces “multipath” issues with thecamera).

Using a multi-module architecture, it is possible to place the depth mapLED or LEDs using the same arrangement as the LEDs for eye tracking butfacing the real world. The same techniques used in eye tracking can beused for illuminating the real world. However, if a more “structuredillumination” is desired, it is possible to have an array of LEDs thatare partially collimated by a reflector. In that case, each LED canilluminate part of the real world and depending on the pattern desired,different LEDs can be activated. Structured illumination can be achievedby means of a partially reflective optical surface that combines acollimating component and a diffractive optical element (DOE) thatcreates the structured illumination pattern.

In the illustrated configuration, the ET illumination is accomplishedwith ET illumination module 208 and ET detection is accomplished with ETdetection module 210. In contrast, depth map module 212 provides bothillumination and detection functionalities in a single module. In asimilar fashion a single ET module could combine the components of ETillumination module 208 and ET detection module 210 into a singlemodule. Such a configuration is described below relative to FIG. 3 andFIG. 11 .

The description above explains that the present concepts allow for pickand match electrical and optical components as modules to achievedesired functionalities, such as RGB display modules, depth sensingmodules, and/or eye tracking modules, among others. These modules can bedistributed and dispersed across the visual assembly 104 so that eachfunctionality is achieved without compromising other functionalities.For instance, the eye tracking modules do not (perceptibly) compromisethe quality of the RGB display perceived by the user. This distributedand dispersed module placement is described in more detail belowrelative to FIG. 3 .

From another perspective, the present concepts offer a pallet ofdifferent components that can be unobstructive or minimally obstructiveto the user so that the user can still see the environment (e.g.,receive ambient visible light from the environment without noticeableinterference). For instance, the electronic components 116 can havedimensions in the x and y reference directions less than 200 microns andin some implementations less than 100 microns, and in someimplementations less than 10 microns. Electronic components of this sizeare so small that they are not visible to the user and are small enoughthat the user tends not to perceive any visual degradation of real-worldimages formed from ambient light 110 passing through the visual assembly104 as long as the components are dispersed rather than clumpedtogether.

Depending on the HMD design parameters, different electronic and/oroptical components can be placed in front of the user across (e.g.,interspersed throughout) the visual assembly 104. These components canachieve various functionalities including: ET detection, ETillumination, monochrome display, RGB/multicolor display, and/or IRdepth sensing, among others, while permitting ambient light to passthrough to the user's eye. The electronic components, given theirdiminutive size may not individually have the emitting or detectingcapabilities of larger (e.g., traditional macroscopic components).However, the components can be operated collectively. For instance,individual electronic devices can contribute to a portion of the eye boxrather than the entire eye box. When analyzed collectively thedistributed arrangement of the electronic components can provide highquality RGB images, eye tracking, and/or 3D mapping, consistent withspecified design parameters.

The visual assembly 104 can be manufactured utilizing varioustechniques. For instance, the electrical layer 108 and the optical layer202 can each be formed individually and then associated with oneanother. The electrical layer 108 can be made on a plastic (e.g., first)substrate with transparent wires (e.g., Indium Tin Oxide (ITO) lines).Using pick and place, different electronic components can be soldered onthis substrate. ITO wires could be used in a “bus arrangement” so thatthe number of electrodes is reduced/minimized.

The optical layer 202 can be used to collimate light, focus, defocusand/or diffuse light. The optical layer can include multiple lenses,mirrors, and/or diffraction elements/components that can be positionedon, and/or formed from, a substrate (e.g., second substrate). Forexample, light from the ET IR LEDs could be partially collimated bymirrors and/or lenses so it more effectively covers the eye box.Alternatively, light from an RGB display could be collimated so it actsas a near eye display. Once completed, an adhesive (not specificallyshown in FIG. 2 ) can be applied to one or both of the electrical layer108 and the optical layer 202 and they can be secured together. Thisconfiguration lends itself to both planar visual assemblies (in the xyreference directions), curved visual assemblies, and visual assemblyimplementations that include both planar regions and curved regions asillustrated in FIG. 2 .

FIG. 3 shows another example HMD device 100B that includes a majorsurface (generally along the xy reference plane) of the visual assembly104. This view shows how the various modules introduced relative to FIG.2 can be distributed and dispersed on the visual assembly 104. In thisimplementation, the ET illumination module 208 and ET detection module210 of FIG. 2 are replaced by a single ET module 302. However, thedescription is equally applicable to the separate and distinct modules208 and 210 described relative to FIG. 2 .

In this configuration, the various modules are placed side-by-side(e.g., adjacent to one another). A majority of the modules can bededicated to generating an RGB image for the user (e.g., RGB displaymodules 206). Other module types can be interspersed with the RGBdisplay modules 206. This interspersing of module types can occur acrossthe entire visual assembly 104 rather than just on the periphery becausethe size of the modules can be small enough that not all modules arerequired to contribute to RGB image generation and the modules do notinterfere perceptibly with RGB light and/or ambient light.

In the illustrated case, modules can be arranged and managed in groupsof seven that approximate a circle as indicated at 302. In this case,five of the seven positions in the circle are occupied by RGB displaymodules 206(1)-206(5). One position is allocated to eye tracking module302(2) and the last position is allocated to depth mapping module212(3). Because of the small size of the modules, this configuration canprovide the same visual experience as if all seven positions wereoccupied by RGB display modules 206. Note that this illustratedconfiguration is provided for purposes of example and many other ratiosof modules can be employed beyond the illustrated 5:1:1 ratio. Forinstance, another implementation can manage a 10×10 array of modules andemploy 98 RGB display modules to one eye tracking module and one depthmapping module, for example.

One aspect of the inventive concepts is the use of an array ofmini-lenses and/or mini-mirrors. Each lens can be used as a miniprojector or a mini camera. This means that traditional eye trackingcameras and traditional eye tracking illuminators can be replaced by agroup of ET modules that are interspersed across the visual assembly,such as among the RGB display modules (e.g., dispersed and distributedamong the RGB display modules) and collectively contribute to the eyetracking functionality. Similarly, a traditional infrared (IR)illuminator for the environment can be replaced by a group of depth mapmodules that are interspersed among the RGB display modules (e.g.,dispersed and distributed among the RGB display modules) andcollectively contribute to a depth mapping functionality.

As mentioned above, one difference between solutions based on thepresent concepts and traditional solutions is the small size (e.g.,visually imperceptible) and the “distributed” nature of the modules.This allows the visual assembly to have more flexibility andsignificantly smaller thickness (e.g., thinner).

FIGS. 4-6 show more details of example HMD devices relating to eyetracking. Eye tracking can be essential in many HMD devices. It can beused to understand the user's interaction with the environment and canbe used as an input device. Many existing HMD devices can use eyetracking to improve image quality as the image is optimized for thespecific location of the user's eye.

There are many existing eye tracking techniques. One of the most commonexisting techniques uses a ring of IR LEDs along the periphery of thevisual assembly. The IR LEDs behave like point sources and emit lighttowards the user's cornea. Light from the cornea is reflected towards acamera. By imaging the reflection of the LEDs, a ring is formed into thecamera and the position of the cornea (and thus of the eye) can bedetermined.

Reflecting LEDs on the cornea works well. However, there is a majordrawback of this traditional technique. The traditional system performsbetter when both the camera and the LEDs are in front of the user. Thisis of course challenging for a VR or AR display where the user shouldnot have any occlusions between their eye and the HMD device's optics.The traditional approach is to bring the ET camera as close to the noseas possible while attaching the LEDs in the rim of the display optics(waveguide or refractive optics). These traditional implementations workwell; however, as the display optics increase in size (for covering alarger FoV) and the display becomes thinner (for ID purposes) the LEDsmove way too close to the eyebrows and cheeks while the camera sees thereflections at a very oblique angle.

The present concepts offer improved performance. As introduced aboverelative to FIGS. 2 and 3 , a potentially key aspect of the inventiveconcepts is the use of many and smaller (e.g., microscopic) lightsources and detectors distributed and dispersed across the visualassembly 104. By using multiple distributed pairs of components tocreate the illumination and detection of the glint, the LEDs anddetectors can be sufficiently small (e.g., less than 100 um) to becomeinvisible to the human eye.

In FIG. 4 the visual assembly 104 of HMD device 100C includes electricallayer 108. A portion of the electrical layer 108 is shown with oneelectronic component 116 positioned in front of the eye 114 in theuser's field of view (FoV). In this case, the electronic component 116is an IR LED 402 that is oriented to emit IR light directly toward theuser's eye 114. This configuration can achieve high efficiency becauseall of the IR light is directed towards the eye box (112, FIG. 1B).

FIG. 5 shows an alternative configuration on HMD device 100D where theelectronic component 116 is manifest as IR LED 402 that is positioned inthe user's FoV. IR LED 402 is oriented to emit IR light away from theuser's eye 114. In this case, optical layer 202 includes opticalcomponent 204 in the form of a partially reflective mirror (e.g., hotmirror) 502. The partially reflective mirror 502 can reflect the IRlight back toward the user's eye 114. The partially reflective mirror502 can have an optical shape that reflects the IR light back toward theuser's eye in a pattern that mimics the IR light being emitted from avirtual point source 504 that is farther from the eye than the visualassembly 104. Thus, the use of the partially reflective mirror 502allows the HMD device 100D to be positioned closer to the user's eyewhile still generating the desired eye tracking IR patterns on theuser's eye 114.

The illustrated configuration directs IR light away from the eye andreflects the IR light from partially reflective mirror (e.g., hotmirror) and towards the eye. While this indirect route may reduceefficiency (as the reflector may be less than 100% efficient) it allowsfor creating a virtual source that may be more convenient for ETpurposes. In addition, multiple lenses can be used to create the samevirtual source but formed by multiple emitters. This aspect is shown inFIG. 6 .

FIG. 6 shows an alternative configuration on HMD device 100E that buildsupon the concepts discussed relative to FIG. 5 . This configurationshows two IR LEDs 402(1) and 402(2) associated with electroniccomponents 116(1) and 116(2), respectively. Note that a discontinuity isshown in the visual assembly 104 to indicate that there can beintervening electronic components and optical components that arediscussed above relative to FIGS. 2 and 3 , but are not shown to avoidclutter on the drawing page.

In this case, the partially reflective (e.g., hot) mirrors 502(1) and502(2) are configured to operate with their respective IR LEDs 402(1)and 402(2) to collectively create an IR image extending toward theuser's eye. For instance, each IR LED and hot mirror pair (e.g., ETillumination module 208) can illuminate a portion of the eye box (112,FIG. 1 ). Stated another way, the partially reflective mirrors 502(1)and 502(2) collectively create an IR image that appears to emanate froma single point source (e.g., virtual point source 504). This singleimage can provide more complete reflection and hence more informationabout a larger portion of the eye (e.g., eye box) than can be achievedwith a single IR LED 402. Alternatively, both IR illumination modulescould be directed to the same portion of the eye box to create a higherintensity IR image at that portion than could be achieved with either IRillumination module alone. In either case, a single ET illuminationmodule 208 is not required to solely illuminate the entire eye box.Higher light intensity can be achieved by focusing individualillumination modules 208 on individual areas of the eye box so thatcollectively the entire eye box is covered with IR light of a desiredintensity, even though none of the individual modules in isolation havesuch capability.

The implementations described above include a single electroniccomponent 116 of a given type, such as LEDs, per optical component 204.Other implementations can have multiple electronic components 116, suchas LEDs associated with individual optical components 204, such aspartially reflective lenses. These LEDs can be controlled in various wayto achieve various functionalities. For instance, all of the LEDs couldbe powered on and off simultaneously for eye tracking illumination toachieve higher IR intensity.

In other cases, the LEDs could be controlled separately. For instance,the LEDs could be powered on and off sequentially. These LEDs can beused; (a) for forming part of a sensing ring of IR LEDs along theperiphery of the visual assembly; and/or (b) be wobbulated so theperformance of the device increases (e.g., increase in resolution ordetermination of other optical properties, like the position on thecornea illuminated). Such a configuration is described below relative toFIGS. 7A-7D.

FIGS. 7A-7D collectively show details relating in inventive conceptsintroduced above. FIG. 7A shows another example HMD device 100F. FIGS.7B-7D show representations of emitted and sensed IR light from the HMDdevice 100F. In this implementation, HMD device 100F can be viewed as ahybrid device that has IR LEDs distributed and dispersed on the visualassembly. IR reflections from the user's cornea 704 can be captured byone or more IR sensors (e.g., cameras) 702 that are positioned aroundthe periphery of the visual assembly 104, such as on the housing 102.

In this configuration, multiple (e.g., three) IR LEDs 402 are positionedin eye tracking module 302. The IR LEDs 402 can have dimensions D in thex and y reference directions of anywhere from 10 microns to 200 micronsand thus are not visible to the user. The IR LEDs 402 can be positionedclose together as indicated by gap G, such as in tens to hundreds ofmicrons apart. The space between the IR LEDs can be occlusive if theirseparation is on the smaller end or transparent if their separation islarger end.

The multiple IR LEDs 402(1)-402(3) can be switched on sequentially orsimultaneously. When switched on sequentially there is less demand onthe spatial response of the IR sensor (e.g., camera) 702 and/or the IRLEDs. When switched on simultaneously there is more demand on thetemporal response of the IR sensor and IR LEDs. In some configurations,such as the wobbulation configuration mentioned above, during a samplingperiod or cycle, each IR LED is activated for a subset of the cycle(e.g., in this example one-third of the cycle). The sensed IRreflections can be analyzed collectively to provide more accurate eyeinformation than can otherwise be obtained.

The three IR LEDs 402 in this example form a simple triangle. Bydetecting the shape of the triangle at the IR sensor 702, otherparameters of the HMD device 100F can be determined. For instance, theseparameters can include the distance between corneal surface 704 and theET module 302 (e.g., between the eye and the electronic components 116).This distance information can also provide information about a localslope of the eye/cornea. While one ET illumination module 208 may, byitself, not allow the IR sensor 702 to provide accurate distance,position, and/or slope information, multiple ET illumination modules 208distributed and disbursed with multiple ET sensing modules can provideinformation sensed by the IR sensor 702 that when analyzed collectivelyis accurate.

FIG. 7B shows a representation of sequential IR emissions 706 from IRLEDs 402(1), 402(2), and 402(3). FIG. 7C shows a representation of theIR detections 708 of the IR emissions 706 as captured by IR sensor 702.FIG. 7D shows a representation of the IR detections 708 superimposed onthe IR emissions 706. The differences or deltas 710 show changes inshape, location, and angular orientation. These changes can be caused bythe user's eye and can provide useful information about the eyelocation, shape, etc. at a resolution greater than would otherwise beachieved.

One example technique for obtaining this higher accuracy eye informationcan utilize the three sequential IR detections 708. The detected imagescan be deconvolved to produce a high-resolution image, even though theindividual images are relatively low resolution. Deconvolution can beused to improve the modulation transfer function (MTF)/point spreadfunction (PSF) of a low-quality optical system. One such technique canemploy multiple IR detectors rather than a single detector. Thecombination of multiple LEDs being controlled and sensed by multipledetectors will provide more accurate information about the eye.

One such example multi-detector is a quadrant detector. Quadrantdetectors have four active photodiode areas defining four quadrants. Thefour active photodiode areas can sense the centroid of an object (e.g.,blob) in the four quadrants. Quadrant detectors operate at highfrequencies, such as mega Hertz frequencies. As such, quadrant detectorscan be used to detect fast eye movement, such as saccades. Someimplementations may employ charge coupled devices (CCDs) orcomplementary metal oxide semiconductors (CMOS) sensors for general IRimaging purposes and quadrant detectors for detecting rapid eyemovements.

The same or similar approach described above can be used to reduce therequirement for the IR sensor 702. For example, by using an IR sensorwith, for example, 10×10 pixels and an IR LED array of (12×12) pixelsthe resolution could be enhanced to approximatelyl 20×120 pixels.Effectively getting N×M super-resolution where N is the number of IRdetectors and M is the number of IR LEDs to get increased resolution ineye position.

The present concepts also provide enhanced pupil imaging for both“bright pupil” (retinal retroreflection) imaging and “dark pupil”imaging. Retinal retroreflection relates to the IR light that reflectsoff the retina straight back toward the source. When the IR sensor isclose to the IR source and both are close to the optical axis, retinalretroreflection is more effective. Due to demographic differences, somepupils are easier to image with dark pupil while some are easier toimage with bright pupil imaging. Bright pupil methods tend to workbetter for some demographics than others. However, dark pupil imagingtends to work better for other demographics. The present concepts canposition IR emitters and IR sensors throughout the optical assemblyincluding proximate to the optical axis. Thus, the present concepts canenable employment of both of these techniques via multiple distributedIR LEDs 402 and multiple IR sensors 702 to achieve accurate eye trackingregardless of the user demographics.

FIG. 8 shows another example HMD device 100G that illustrates that an IRsensor 702 can sense a portion of the eye box via partially reflectivemirror 502. The partially reflective mirror 502 can function as the IRsensor's lens in the illustrated configuration. The IR sensor 702 can bea single IR sensor, or multiple IR sensors. For instance, the detectorcould be an IR photodetector array. The use of multiple IR sensorsoperating cooperatively can provide higher resolution data than a singlesensor as described above and below.

The illustrated IR sensor 702 can sense an individual portion of the eyebox while other IR sensors sense other portions of the eye box. FIG. 9illustrates this aspect. In FIG. 9 , HMD device 100H is similar to HMDdevice 100G except that two IR sensors 702 are illustrated with twopartially reflective mirrors 502. The orientation of individualpartially reflective mirrors 502 can be adjusted so that each IR sensorand partially reflective mirror pair senses a different part of the eyebox. This difference in orientation causes IR sensor 702(1) to receiveIR light at angles one and two and IR sensor 702(2) to receive IR lightat angles three and four. The entire eye box can be sensed byintegrating the data from the various IR sensors. While only two IRsensors and partially reflective mirror pairs are illustrated, hundredsor thousands of pairs may be employed.

FIG. 10 shows another example HMD device 100I that illustrates how thepresent implementations can enable both bright and dark pupil imagingsimultaneously using distributed and dispersed IR LEDs 402 and IRsensors 702. In this configuration multiple IR LEDs can emit light thatupon reflection can be sensed by one or more of the IR sensors 702. Atthe illustrated point in time, IR LED 402(3) emitted light thatreflected back from the user's retina and is sensed by IR sensor 702(3)and can be processed consistent with bright pupil techniques. Meanwhile,IR light from IR LED 402(1) could be sensed by IR sensor 702(1) andprocessed consistent with dark pupil techniques. Finally, the IR lightfrom IR LED 402(5) is sensed by IR sensor 702(5) and can be processedcollectively with information from IR sensor 702(1). This combination ofemission and detection from IR LEDs and sensors interspersed across thevisual assembly 104 can ensure accurate eye tracking throughout the userexperience. This can occur even if some of them are blocked or do notprovide determinative data because of eye color issues, eye lidposition, etc.

Note that for ease of explanation, the electronic components of theelectrical layer 108 have generally been illustrated in a single layer,such as an IR sensor 702 adjacent to an IR LED 402 along the xyreference plane. However, other implementations can stack electroniccomponents in the z direction. One such example is described below inrelation to FIG. 11 .

FIG. 11 shows another example HMD device 100J. In this case, IR LED 402and IR sensor 702 are stacked in the z reference direction (e.g.,parallel to the optical axis) on the electrical layer 108. In thisconfiguration, the IR LED 402(1) is emitting light toward the eye 114.IR light reflected back from the eye is focused by partially reflectivemirror 502(1) onto IR sensor 702(1). Similarly, the IR LED 402(2) isemitting light toward the eye 114. IR light reflected back from the eyeis focused by partially reflective mirror 502(2) onto IR sensor 702(2).The electronic components may tend to obstruct ambient visible lightfrom the environment more than those areas of the electrical layerwithout electronic components. Thus, stacking electronic componentstends to increase the ratio of unobstructed areas to potentially orsomewhat obstructed areas as indicated on FIG. 11 .

In the same way that IR LEDs 402 can direct IR illumination towards theuser, the IR sensor 702 may be configured to image a particular area ofthe eye box. Because of the simplicity of optics (a single reflector vsmultiple refractive elements in an ET camera) the FoV of the IR sensorcan be relatively small to reduce aberrations.

As mentioned, the field of view of each IR sensor 702 can be less than atraditional sensor positioned on the housing. This is not an issuebecause data from multiple sensors can be used to collectively capturethe entire eye box. Note that in practice the FoV of the two (or more)lenses may require some overlap. This is because the lenses are not atinfinity compared to the position of the eye and thus the potential needto capture a wider FoV per lens.

It is also possible to combine the use of the IR LED 402 and IR sensor702 in a single lenslet. This configuration can minimize the occlusionscaused as the LED and sensor occupy the same space. It may also bringsome advantages in terms of geometry as the source and detector will beat the same point.

Note also that the present concepts offer many implementations. Forinstance, in HMD device 100J of FIG. 11 , the IR LEDs 402 face towardthe eye and the IR sensors 702 face the opposite way and receive IRlight that is reflected from the user's eye and reflected again by thepartially reflective mirror 502. Alternatively, the components could beswapped so that the IR LEDs 402 could emit toward the partiallyreflective mirrors 502. IR light reflected by the partially reflectivemirrors 502 and again off of the user's eye could be detected by IRsensors 702 (potentially with the aid of a small lens, which is notspecifically shown).

The same or similar arrangements can work with a transmissive or acombination of transmissive and reflective optical components. Inaddition, other optical components (diffractive, holographic,meta-optics) could be employed.

Consistent with the present implementations various coatings can beemployed on the partially reflective mirrors 502 when ET and depthsensing IR illumination is used. For instance, the coatings can bedielectrics and tuned to a particular wavelength. That can improve thetransparency of the combiner when used in an AR system.

It is also possible to combine the functions of ET, depth sensing andRGB display in a single element. This aspect is discussed in more detailbelow relative to FIG. 15 .

FIG. 12 shows an alternative arrangement to FIG. 11 . In this case, theexample HMD device 100K positions the IR LEDs 402 away from the eye 114.IR light emitted by the IR LEDs 402 is reflected back toward the eye bythe partially reflective mirrors 502. IR light that reflects from theeye is focused by lenses 1002 onto the IR sensors 702.

The discussion above relative to FIGS. 2 and 3 explains that the presentdistributed and dispersed module concepts can be applied to eye trackingand depth sensing among other functionalities. FIGS. 4-12 explaindetailed configurations of multiple implementations relating to eyetracking. Those details are also applicable to depth sensing. One suchexample is shown relative to FIG. 13 .

FIG. 13 shows another example HMD device 100L that can provide depthsensing on visual assembly 104. In this case the IR LEDs 402 are facingtoward the eye 114. Partially reflective mirrors 502 on the opticallayer 202 are oriented to reflect the IR light back toward theenvironment (e.g., away from the eye 114) as if the IR light was emittedfrom virtual point source 504 on the eye side. IR light reflected backfrom the environment can be detected by IR sensors 702, such as CMOSstereoscopic depth sensors, among others.

FIG. 14 shows another example HMD device 100M that can provide eyetracking and can generate color images to the user's eye 114. In thiscase, multiple LEDs 1402 are distributed across the electrical layer108. In this configuration, all of the LEDs are the same in that theyemit light with the same properties. A determinative layer 1404, such asa quantum dot matrix is positioned relative to the LEDs. Thedeterminative layer can have localized differences that affect the lightthat passes through the determinative layer from the individual LEDs1402. For instance, the determinative layer 1404 can cause emitted lightfrom LED 1402(1) and 1402(11) to be IR (T, on the FIGURE) light, whilelight from the remaining LEDs can be dedicated to visible RGB light.Stated another way, the electronic components (e.g., the LEDs 1402) canbe generic for multiple modules of the electrical layer 108. Thedeterminative layer 1404 positioned over individual modules can define afunctionality of the module, such as the wavelength(s) of light emittedby the module.

In some cases, the IR light can be uniformly emitted across the visualassembly 104 (e.g., a ratio of IR emitters to RGB emitters can beuniform across the visual assembly). In other cases, the ratios ofvisible light and IR light may be different for different regions of thevisual assembly 104.

In one such example of the latter configuration, visible light may beproduced in higher concentrations proximate to the optical axis (e.g.,less IR light) for enhanced image quality. Further from the opticalaxis, a percentage of IR light to RGB light can increase. Stated anotherway, the ratio of RGB emitters to IR emitters can be higher proximate tothe optical axis and lower farther from the optical axis. The user tendsto look along the optical axis and foveal regions along the user's lineof sight can have a higher concentration of RGB light output to providea higher possible image quality that can be offered by high RGB density.Further from the optical axis, the user's visual acuity tends to be lessand more resources can be dedicated to eye tracking without affectingthe perceived image quality. In some device configurations, the IR/RGBratios can be static (e.g., unchangeable). Other device configurationcan offer dynamically adjustable ratios. For instance, the initialconfigurations can be dynamically changed in some configurations, suchas if the eye tracking indicates the user is looking to the side ratherthan straight in front. Such an example device configuration isdescribed relative to FIG. 15 .

FIG. 15 shows another example HMD device 100N that can provide eyetracking and can generate color images to the user's eye 114. Thisexample includes multiple adjacent (but potentially spaced apart)modules 1502 on the visual assembly 104. In this case, each module 1502includes an LED light source that can produce IR and RGB light. Eachmodule 1502 also includes a light detector that can detect IR and/orvisible light. Each of these modules can be powered and/or controlledvia one or more conductors (not specifically designated) in the visualassembly. Individual modules 1502 can be dynamically controlled tocontribute to RGB images, eye tracking, or powered off, depending uponvarious parameters, such as eye gaze direction and/or foveation, amongothers. In some cases, the module may contribute to image generation foran entire cycle of image generation (e.g., frame). In other cases, themodule may contribute to image generation for a sub-cycle of imageduration and contribute to another function, such as eye tracking duringanother sub-cycle. Alternatively, the functionality may change if theuser looks toward or away from the individual module.

The discussion above emphasizes emitting visible light or IR light,however, the LEDs 1402 can be controlled to selectively emit one or moreof several IR wavelengths. This can allow different properties of eachwavelength to be leveraged depending on the conditions and/or function.For instance, some wavelengths can provide better directionalsensitivity than others to determine where the light is coming from.Further, different wavelengths can help with imaging the eye. Forexample, retinal images can be enhanced by using different wavelengths.Utilizing multiple IR wavelengths can facilitate distinguishing retinalreflections from corneal reflections. Conditions can also influencewhich IR wavelengths to utilize. For instance, some IR wavelengths aremore affected by environmental factors. For example, 940 nm wavelengthIR light is less affected by sunlight than lower wavelength IR light.Thus, 940 nm wavelength IR light could be employed outside in brightconditions and 830 nm wavelength IR light could be employed in lowerlight conditions, such as indoor environments.

FIG. 16 shows a system 1600 that includes HMD device 100P that issimilar to HMD device 100 described above relative to FIGS. 1A and 1B.As introduced above, the HMD device 100P can include housing (e.g.,frame) 102 that positions the visual assembly 104 in line with theuser's eye 114 along the optical axis. The electrical layer 108 and/orthe optical layer 202 can include multiple microscopic (e.g., invisibleto the user) components that are distributed across the visual assemblyin the user's and/or the device's FoV including along the optical axis.The components can operate as modules that achieve specific functions,such as eye tracking, 3D sensing, and/or RGB image generation yet areimperceptible to the user.

The HMD device 100P can also include a controller 1602, a processingunit 1604, storage and/or memory 1606, a communication unit 1608, and/ora power supply 1610. In some implementations controller 1602 may includethe processing unit 1604 and the memory 1606. The controller can utilizethe memory for storing processor readable instructions and/or data, suchas user data, image data, sensor data, etc. The communication unit 1608can be communicatively coupled to the processing unit 1604 and can actas a network interface for connecting the HMD device to another computersystem represented by computer 1612. The computer 1612 may includeinstances of any of the controller 1602, processing units 1604, memory1606, communication units 1608, and power supplies 1610. The HMD device100P may be robust and operate in a stand-alone manner and/or maycommunicate with the computer 1612, which may perform some of thedescribed functionality.

Controller 1602 may provide commands and instructions, such as drivingpower to the electronic components 116 to generate visible and/ornon-visible light. Similarly, the controller can receive data fromsensors, such as IR sensors 702. The controller can use the data toidentify information about the eye (e.g., eye tracking) and/or theenvironment (e.g., 3D mapping).

The controller 1602 can analyze the data from the sensors to identifyfeatures of the cornea and/or retina, such as by detecting glints oflight and/or other detectable features associated with the user's eye,to determine the pupil position and gaze direction of the eye.

The storage/memory 1606 can include an optics model 1614 and/or measuredperformance (e.g., deviation data) 1616. The optics model 1614 can bederived from the design specifications of the HMD device and thedistributed and dispersed arrangement of the various modules. Recallthat the eye information from any individual eye tracking module or 3Dmapping module may not be as robust as traditional designs positionedoutside the FoV. The controller can analyze the eye informationcollectively to identify meaningful eye information.

The controller 1602 can use this eye information to control the modules.For instance, the controller may increase image resolution generated byRGB LEDs in foveated regions and decrease image resolution outside thefoveated regions. Similarly, the controller can use eye movement toincrease resolution in regions of the visual assembly the eyes aremoving toward and decrease resolution in regions the eyes are movingaway from.

In some implementations, the controller 1602 may also employ artificialintelligence algorithms, such as neural networks, for analyzing sensordata from the distributed sensors. The data from any one sensor may berather rudimentary, yet the artificial intelligence algorithms cancollectively analyze data from the available sensors to find meaningfulpatterns that are not apparent with traditional analytics.

Processing unit 1604 may include one or more processors including acentral processing unit (CPU) and/or a graphics processing unit (GPU).Memory 1606 can be a computer-readable storage media that may storeinstructions for execution by processing unit 1604, to provide variousfunctionality to HMD device 100P. Finally, power supply 1610 can providepower for the components of controller 1602 and the other components ofHMD device 100P.

The term “device”, “computer,” “computing device,” “client device,”“server,” and/or “server device” as used herein can mean any type ofdevice that has some amount of hardware processing capability and/orhardware storage/memory capability. Processing capability can beprovided by one or more hardware processing units 1604 and/or otherprocessors (e.g., hardware processing units/cores) that can executecomputer-readable instructions to provide functionality.Computer-readable instructions and/or data can be stored on persistentstorage or volatile memory. The term “system” as used herein can referto a single device, multiple devices, etc.

Memory 1606 can be storage resources that are internal or external toany respective devices with which it is associated. Memory 1606 caninclude any one or more of volatile or non-volatile memory, hard drives,flash storage devices, and/or optical storage devices (e.g., CDs, DVDs,etc.), among others. As used herein, the term “computer-readable media”can include signals. In contrast, the term “computer-readable storagemedia” excludes signals. Computer-readable storage media includes“computer-readable storage devices.” Examples of computer-readablestorage devices include volatile storage media, such as RAM, andnon-volatile storage media, such as hard drives, optical discs, andflash memory, among others, which may constitute memory 1606.

In some cases, the HMD devices are configured with a general-purposehardware processor and storage resources. In other cases, a device caninclude a system on a chip (SOC) type design. In SOC designimplementations, functionality provided by the device can be integratedon a single SOC or multiple coupled SOCs. One or more associatedprocessors can be configured to coordinate with shared resources, suchas memory, storage, etc., and/or one or more dedicated resources, suchas hardware blocks configured to perform certain specific functionality.Thus, the term “processor,” “hardware processor” or “hardware processingunit” as used herein can also refer to central processing units (CPUs),graphical processing units (GPUs), holographic processing units (HPUs),controllers, microcontrollers, processor cores, or other types ofprocessing devices suitable for implementation both in conventionalcomputing architectures as well as SOC designs.

Alternatively, or in addition, the functionality described herein can beperformed, at least in part, by one or more hardware logic components.For example, and without limitation, illustrative types of hardwarelogic components that can be used include Field-programmable Gate Arrays(FPGAs), Application-specific Integrated Circuits (ASICs),Application-specific Standard Products (ASSPs), System-on-a-chip systems(SOCs), Complex Programmable Logic Devices (CPLDs), etc.

In some configurations, any of the code discussed herein can beimplemented in software, hardware, and/or firmware. In any case, thecode can be provided during manufacture of the device or by anintermediary that prepares the device for sale to the end user. In otherinstances, the end user may install these code later, such as bydownloading executable code and installing the executable code on thecorresponding device.

Also note that the components and/or devices described herein canfunction in a stand-alone or cooperative manner to implement thedescribed techniques. For example, the methods described herein can beperformed on a single computing device and/or distributed acrossmultiple computing devices that communicate over one or more network(s).Without limitation, such one or more network(s) can include one or morelocal area networks (LANs), wide area networks (WANs), the Internet, andthe like.

Example Methods

FIG. 17 illustrates an example method 1700, consistent with the presentconcepts. Method 1700 can be implemented by a single device, e.g., HMDdevice 100, or various steps can be distributed over one or moreservers, client devices, etc. Moreover, method 1700 can be performed byone or more components, such as a controller and/or by other componentsand/or devices.

At block 1702, the method can operate non-visible light emitters andsensors distributed across a transparent visual assembly of an HMDdevice with visible light emitters.

At block 1704, the method can identify properties of an eye of a userwearing the HMD device based at least in part from data from thenon-visible light sensors.

At block 1706, the method can update operation of at least one of thenon-visible light emitters and sensors or the visible light emittersbased at least in part upon the properties of the eye of the useridentified from the data from the non-visible light sensors.

Various examples are described above. Another example relates to an eyetracking system where the illumination is placed on a see-throughtransparent substrate (e.g., combiner) and directed towards the users'eye.

Another example includes an eye tracking system where the illuminationis placed on a see-through transparent substrate and pointed towards thereal world. A reflector (e.g., IR selective reflector or partial mirror)collimates or partially collimates the LED illumination towards an eyebox of an HMD device.

Another example taken alone or in combination with any of the above orbelow examples includes an eye tracking system where multiple LEDs areplaced on a see-through transparent substrate and pointed towards thereal world. A different type reflector is used for each LED so an entireeye box is illuminated by combining the illumination from multiple LEDs.

Another example taken alone or in combination with any of the above orbelow examples includes an eye tracking system where the IR lightdetector (camera or a single detector) is using a reflector embeddedinto the combiner to collimate and focus the beam on the detector.

Another example taken alone or in combination with any of the above orbelow examples includes an eye tracking system where both bright anddark images are imaged simultaneously.

Another example taken alone or in combination with any of the above orbelow examples includes an eye tracking system that uses multiplewavelengths.

Another example taken alone or in combination with any of the above orbelow examples includes an eye tracking system where multiple IR lightdetectors (camera or a single detector) are using different typereflectors embedded into the combiner to collect light from differentparts of the eye box, and focus it on the detectors.

Another example taken alone or in combination with any of the above orbelow examples includes a system where the reflector is combined withother non-reflective optics.

Another example includes an eye tracking system where there is aplurality of LED (or display pixels) and detector (or camera pixel)arrays. Each LED or detector array faces an embedded reflector thatcollimates the outcoming or incoming light to or from the eye box. Bycombining multiple LEDs and detectors an improvement in resolution canbe achieved.

Another example taken alone or in combination with any of the above orbelow examples includes an eye tracking system where there is aplurality of LED (or display pixels) and detector (or camera pixel)arrays. Each LED or detector array faces an embedded reflector thatcollimates the outcoming or incoming light to or from the eye box. Bycombining multiple LEDs and detectors an improvement in resolution canbe achieved. Each LED or detector is activated at a different time sotemporal resolution can be used to improve spatial resolution of the ETsystem.

Another example taken alone or in combination with any of the above orbelow examples includes an ET system where each LED source is composedof a number of sub-elements/pixels. By imaging these pixels on an ETcamera and measuring the distortion of the IR pattern, more informationcan be obtained about the reflective surface (i.e., cornea).

Another example taken alone or in combination with any of the above orbelow examples includes a depth sensing system (such as Time of flight)where the “flood illumination” LEDs are attached on the combiner of thedisplay and point directly towards the real world.

Another example taken alone or in combination with any of the above orbelow examples includes a depth sensing system (Time of flight orstereo) where the “flood illumination” LEDs are attached on the combinerof the display and point directly towards the user and then arereflected to the real world by an IR/partial mirror. This allows for thebeam to have specific profile when illuminating the real world.

Another example taken alone or in combination with any of the above orbelow examples includes a depth sensing system (Time of flight orstereo) where an array of illumination LEDs are attached on the combinerof the display and point directly towards a reflector and then reflectedto the real world by an IR/partial mirror. By switching differentLEDs/pixels ON/OFF, it is possible to create a structured illuminationthat can enable or enhance depth sensing.

Another example taken alone or in combination with any of the above orbelow examples includes a depth sensing system where the camera isembedded into the combiner of the HMD device.

Another example taken alone or in combination with any of the above orbelow examples includes a depth sensing system where multiple camerasare embedded into the combiner of the HMD device. Each camera can coverpart of the environment with different resolution or FoV.

Another example includes an HMD device that uses a plurality ofmini-lenses to create the virtual image into the user's eye. Such asystem can contain lenses that (a) form the image into the user's eye(b) enable ET by the use of emitters and sensors embedded into the minilenses (c) facilitate or enhance depth sensing by providing lenses thatemit light into the environment or sensors that collect light from theenvironment.

Another example includes a head mounted display device comprising ahousing configured to be positioned relative to a head and eye of a userand a visual assembly positioned by the housing in front of the user'seye, the visual assembly comprising an electrical layer comprisingside-by-side electronic components, individual electronic componentsconfigured to emit or detect light and an optical layer comprisingside-by-side optical components, individual optical componentsconfigured to refract or reflect or diffract light relative toindividual electronic components.

Another example can include any of the above and/or below examples wherethe electrical layer and the optical layer are formed on a singlesubstrate or wherein the electrical layer comprises a first substrateand the optical layer comprises a second substrate, and wherein thefirst and second substrates are positioned against one another orwherein the first and second substrates are spaced apart from oneanother.

Another example can include any of the above and/or below examples wherethe optical layer is transparent.

Another example can include any of the above and/or below examples whereat least some of the electronic components and optical componentscontribute to eye tracking of the eye of the user.

Another example can include any of the above and/or below examples wherethe electrical layer is positioned proximate to the user relative to theoptical layer.

Another example can include any of the above and/or below examples whereindividual electronic components are paired with individual opticalcomponents as modules to achieve specific functionalities.

Another example can include any of the above and/or below examples wherethe specific functionalities include eye tracking illumination, eyetracking detection, image generation, 3D illumination, and/or 3Ddetection.

Another example can include any of the above and/or below examples wherean individual eye tracking illumination pair comprises an individualelectronic component that emits non-visible light away from the user'seye and an individual optical component that redirects the non-visiblelight back towards the user's eye.

Another example can include any of the above and/or below examples wherean individual eye tracking detection pair further comprises a lens thatreceives the non-visible light reflected from the user's eye and focusesthe non-visible light toward another individual electronic componentthat senses the non-visible light reflected back from the user's eye.

Another example can include any of the above and/or below examples wherethe another electronic component faces the user's eye or wherein theanother electronic component is positioned behind the electroniccomponent.

Another example can include any of the above and/or below examples whereeye tracking illumination pairs and individual eye tracking detectionpairs are distributed across the visual assembly.

Another example includes a head mounted display device comprising ahousing configured to be positioned relative to a head and eye of a userand a transparent visual assembly positioned by the housing in front ofthe user's eye and comprising multiple eye tracking illuminatorsdistributed across the transparent visual assembly and configured toemit non-visible light and multiple eye tracking detectors distributedacross the transparent visual assembly and configured to detect thenon-visible light reflected back from the eye of the user.

Another example can include any of the above and/or below examples wherethe eye tracking illuminators are configured to emit the non-visiblelight in a direction away from the eye of the user.

Another example can include any of the above and/or below examples wherethe transparent visual assembly further comprises optical componentsthat include non-visible selective reflectors that are configured tocollimate the non-visible light in an eye box defined by the headmounted display device.

Another example can include any of the above and/or below examples wherethe optical components are configured to operate cooperatively toilluminate an entire eye box for the user.

Another example can include any of the above and/or below examples whereother optical components are distributed across the transparent visualassembly and configured to cooperatively generate a visual image in theeye box.

Another example can include any of the above and/or below examples whereother optical components are configured to generate the visual imagesimultaneously to the optical components illuminating the entire eye boxwith the non-visible light.

Another example can include any of the above and/or below examples wherethe optical components, the other optical components, and the additionaloptical components are interspersed across a field of view of thetransparent visual assembly.

Another example can include any of the above and/or below examples wherethe eye tracking illuminators are configured to emit the non-visiblelight in a direction toward the eye of the user.

Another example comprises a system that includes a visual assemblyconfigured to be positioned in front of an eye of a user and comprisingmultiple eye tracking illuminators distributed across the visualassembly and configured to emit non-visible light and multiple eyetracking detectors distributed across the visual assembly and configuredto detect the non-visible light reflected back from the eye of the userand a controller configured to process the detected non-visible lightfrom multiple eye tracking detectors to identify information relating tothe eye.

Another example can include any of the above and/or below examples wherethe controller is located on an HMD device that includes the visualassembly or wherein the controller is located on a computer that isconfigured to communicate with the HMD device.

CONCLUSION

Although the subject matter has been described in language specific tostructural features and/or methodological acts, it is to be understoodthat the subject matter defined in the appended claims is not limited tothe specific features or acts described above. Rather, the specificfeatures and acts described above are disclosed as example forms ofimplementing the claims and other features and acts that would berecognized by one skilled in the art are intended to be within the scopeof the claims.

1. A head mounted display device, comprising: a housing configured to bepositioned relative to a head and eye of a user; and, a visual assemblypositioned by the housing in front of the user's eye, the visualassembly comprising: an electrical layer comprising side-by-sideelectronic components, individual electronic components configured toemit or detect light; and, an optical layer comprising side-by-sideoptical components, individual optical components configured to refractor reflect or diffract light relative to individual electroniccomponents.
 2. The head mounted display device of claim 1, wherein theelectrical layer and the optical layer are formed on a single substrateor wherein the electrical layer comprises a first substrate and theoptical layer comprises a second substrate, and wherein the first andsecond substrates are positioned against one another or wherein thefirst and second substrates are spaced apart from one another.
 3. Thehead mounted display device of claim 2, wherein the optical layer istransparent to visible light.
 4. The head mounted display device ofclaim 1, wherein at least some of the electronic components and opticalcomponents contribute to eye tracking of the eye of the user.
 5. Thehead mounted display device of claim 1, wherein the electrical layer ispositioned proximate to the user relative to the optical layer.
 6. Thehead mounted display device of claim 1, wherein individual electroniccomponents are paired with individual optical components as modules toachieve specific functionalities.
 7. The head mounted display device ofclaim 6, wherein the specific functionalities include eye trackingillumination, eye tracking detection, image generation, 3D illumination,and/or 3D detection.
 8. The head mounted display device of claim 7,wherein an individual eye tracking illumination pair comprises anindividual electronic component that emits non-visible light away fromthe user's eye and an individual optical component that redirects thenon-visible light back towards the user's eye.
 9. The head mounteddisplay device of claim 8, wherein an individual eye tracking detectionpair further comprises a lens that receives the non-visible lightreflected from the user's eye and focuses the non-visible light towardanother individual electronic component that senses the non-visiblelight reflected back from the user's eye.
 10. The head mounted displaydevice of claim 9, wherein the another electronic component faces theuser's eye or wherein the another electronic component is positionedbehind the electronic component.
 11. The head mounted display device ofclaim 10, wherein eye tracking illumination pairs and eye trackingdetection pairs are distributed across the visual assembly.
 12. A headmounted display device, comprising: a housing configured to bepositioned relative to a head and eye of a user; and, a transparentvisual assembly positioned by the housing in front of the user's eye andcomprising multiple eye tracking illuminators distributed across thetransparent visual assembly and configured to emit non-visible light andmultiple eye tracking detectors distributed across the transparentvisual assembly and configured to detect the non-visible light reflectedback from the eye of the user.
 13. The head mounted display device ofclaim 12, wherein the eye tracking illuminators are configured to emitthe non-visible light in a direction away from the eye of the user. 14.The head mounted display device of claim 13, wherein the transparentvisual assembly further comprises optical components that includenon-visible selective reflectors that are configured to collimate thenon-visible light in an eye box defined by the head mounted displaydevice.
 15. The head mounted display device of claim 14, wherein theoptical components are configured to operate cooperatively to illuminatean entire eye box for the user.
 16. The head mounted display device ofclaim 15, further comprising other optical components distributed acrossthe transparent visual assembly and configured to cooperatively generatea visual image in the eye box.
 17. The head mounted display device ofclaim 16, wherein the other optical components are configured togenerate the visual image simultaneously to the optical componentsilluminating the entire eye box with the non-visible light.
 18. The headmounted display device of claim 17, further comprising additionaloptical components that are configured to three-dimension (3D) map aregion in front of the user simultaneously to the other opticalcomponents generating the visual image and the optical componentsilluminating the entire eye box with the non-visible light.
 19. The headmounted display device of claim 18, wherein the optical components, theother optical components, and the additional optical components areinterspersed across a field of view of the transparent visual assembly.20. A system, comprising: a visual assembly configured to be positionedin front of an eye of a user and comprising multiple eye trackingilluminators distributed across the visual assembly and configured toemit non-visible light and multiple eye tracking detectors distributedacross the visual assembly and configured to detect the non-visiblelight reflected back from the eye of the user; and, a controllerconfigured to process the detected non-visible light from multiple eyetracking detectors to identify information relating to the eye.